Skip to content

[trainer throughput] Cap Nemotron-3.5-Lightning 64K micro-batches at 49,152 tokens - #247

Merged
Micah Yong (micahtyong) merged 1 commit into
mainfrom
micah/nemotron-perf/trainer-4-micro-batch-49152
Nov 17, 2026
Merged

Micah Yong (micahtyong) merged 1 commit into
mainfrom
micah/nemotron-perf/trainer-4-micro-batch-49152

Conversation

@micahtyong

@micahtyong Micah Yong (micahtyong) commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Merge after #257 and the previous PR in this stack. This change only pays together with the previous PR, which needs #257.

Summary

max_tokens_per_gpu sets how many tokens the trainer packs into one micro-batch. At 65,536, a batch of 16K-token rows fills each micro-batch to the limit. With all experts on every GPU (the previous PR), those full micro-batches push reserved GPU memory to 132 GB of 140 GB, and the step slows down as the allocator runs out of room.

This PR lowers max_tokens_per_gpu to 49,152 in nemotron35_lightning_30b_a3b_lora_64k. A row longer than that still trains as its own micro-batch, so 64K-token rows work as before. The 256k recipe keeps 65,536.

E2E results

Setup: 2x H200, one client, stock Tinker SDK, LoRA rank 32. Short rows are 128 rows of about 1,600 tokens per step, long rows are 8 rows of about 16,000 tokens, and 64K rows are 2 rows of 64K tokens. Per-token log probabilities match the previous PR to within run-to-run noise.

Batch Before WandB After WandB forward_backward (s)
Short rows Before After 9.06 → 9.03
Long rows Before After 6.79 → 5.52
64K rows Before After 6.54 → 6.56

@micahtyong
Micah Yong (micahtyong) force-pushed the micah/nemotron-perf/trainer-4-micro-batch-49152 branch from d7e6542 to d335cac Compare November 17, 2026 02:22
@micahtyong
Micah Yong (micahtyong) force-pushed the micah/nemotron-perf/trainer-3-expert-parallel-1 branch from 266c2f3 to edc6695 Compare November 17, 2026 02:33
@micahtyong
Micah Yong (micahtyong) changed the base branch from micah/nemotron-perf/trainer-3-expert-parallel-1 to main November 17, 2026 02:46
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@micahtyong
Micah Yong (micahtyong) force-pushed the micah/nemotron-perf/trainer-4-micro-batch-49152 branch from 1a29a48 to c8c8e9f Compare November 17, 2026 02:58
@micahtyong
Micah Yong (micahtyong) marked this pull request as ready for review November 17, 2026 03:33
@micahtyong
Micah Yong (micahtyong) merged commit 7f2d9ba into main Nov 17, 2026
3 checks passed
@micahtyong
Micah Yong (micahtyong) deleted the micah/nemotron-perf/trainer-4-micro-batch-49152 branch November 17, 2026 03:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant